Deep Convolution Neural Networks, or Deep CNN, is the core of the remarkable development in the field of Deep Learning. Though the CNN has been employed to solve character recognition problems in 1990s, it is not until recently that the CNN has become widespread in Machine Learning. Due to the recent researches, the CNN has been a very useful and powerful tool in the field of Machine Learning. For example, in 2012, the CNN significantly outperformed its competitors in an annual software contest, the ImageNet Large Scale Visual Recognition Challenge, and won the contest. After that, the CNN has become a very useful tool in the field of machine learning.
FIG. 1 shows a configuration of a conventional learning device with the CNN capable of performing a CReLU operation and a convolution operation.
By referring to FIG. 1, the conventional learning device has many convolutional layers, e.g., convolutional layers 110 and 130, and many Concatenated Rectified Linear Units (CReLU), e.g., a CReLU 120 and other CReLUs (not illustrated), which are disposed alternately.
Herein, the CReLU 120 may include two Rectified Linear Units (ReLUs) 122 and operations of the ReLUs 122 are performed by following Formulas:ReLU(x)=max(0,x)  [Formula 1]
The CReLU is an activation scheme that additionally makes negative activation as well as positive activation of the ReLU just as following Formula.CReLU(x)=(ReLU(x),ReLU(−x))=(max(0,x),max(0,−x))  [Formula 2]
In comparison of the ReLU, the CReLU may double the number of result values without an increase of computation load of the convolutional layers. Accordingly, the CReLU may increase a detection accuracy while maintaining a detection speed.
In FIG. 1, the CReLU 120 includes one scale layer 121, the two ReLU layers 122, and a concatenation layer 123.
In detail, the scale layer 121 multiplies a value outputted from a previous convolutional layer 110 by a scale value of −1, resulting in a scaled value.
Further, one of the ReLU layers 122 receives the value outputted from the previous convolutional layer 110 and then applies ReLU operation to the received value and the other one of the ReLU layers 122 receives the scaled value from the scale layer 121 and then applies ReLU operation to the scaled value.
By referring to FIG. 1, if
         (                            1                                      -            2                                                            -            3                                    4                      )  (i.e., a feature map whose pixel value is 1 at x=0, y=0, and channel ch=0, whose pixel value is 2 at x=1, y=0, and ch=1, whose pixel value is 3 at x=0, y=1, and ch=1, whose pixel value is 4 at x=1, y=1, and ch=0 and whose pixel values are 0 except for the above-mentioned four pixels) as the value outputted from the previous convolutional layer 110 is inputted to the CReLU 120, a first ReLU layer receives
         (                            1                                      -            2                                                            -            3                                    4                      )  and a second ReLU layer receives
         (                                        -            1                                    2                                      3                                      -            4                                )  and then each of the first ReLU layer and the second ReLU layer performs positive activation operations to thereby generate
         (                            1                          0                                      0                          4                      )  and
      (                            0                          2                                      3                          0                      )    ,which are then delivered to the concatenation layer 123. Then, the concatenation layer 123 concatenates
         (                            1                          0                                      0                          4                      )  and
         (                            0                          2                                      3                          0                      )  and thus generates
         [                  (                                            1                                      0                                                          0                                      4                                      )            ⁢              (                                            0                                      2                                                          3                                      0                                      )              ]  and then delivers
         [                  (                                            1                                      0                                                          0                                      4                                      )            ⁢              (                                            0                                      2                                                          3                                      0                                      )              ]  to the convolutional layer 130. The convolutional layer 130 performs convolution operations to
      [                  (                                            1                                      0                                                          0                                      4                                      )            ⁢              (                                            0                                      2                                                          3                                      0                                      )              ]    .
As shown in FIG. 1, if an operation of the CReLU layer and that of the convolutional layer are separately performed, each of independent operations is performed and therefore, the computational load becomes too much.
Accordingly, the inventors of the present invention provide a novel method for reducing the computational load of a CNN where CReLU is used.